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ABSTRACT : 

Two formulae are derived for ratios of limiting probabilities for 
a class of finite homogeneous Markov chains. The class consists of chains 
obtained by generalization of Bernoulli random walk with reflecting or 
absorbing barriers. These chains are closely related to problems of 
testing hypotheses with finite memory. The formulae are recursive in 
nature and hence much easier to use than classical methods. 
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1. Introduction. 



In this report we propose a method for recursive evaluation of 
certain probabilities associated with two classes of finite homogeneous 
Markov chains. These chains are next of kin to Bernoulli random walks 
with reflecting and absorbing barriers. They may be roughly character- 
ized by the following four properties: 

(1) Except for the barrier states, (one-step) transitions from each 
state can be made to exactly two other states. 

(2) The states are divided into two subsets, and S , plus an 

initial state in the absorbing case. States in S communicate 
with states in only via a pair of states, one in each subset. 

(3) Except for the absorbing states, the probability of transition 

among states in and among states in has only two values, 

p^, 1 - p^ and p^ , 1 - p^ respectively. 

(4) If states in each of the two subsets are ordered such that the 
transitions with probability p are all between adjacent states in 
one direction then the transitions with probability 1 - p are all 
in the opposite direction, but not necessarily to the adjacent state. 

A glance at Figures 1 and 3 may help to reveal the structure of a typical 

member of those two classes. 

We are interested in the ratio of limiting probabilities of a 

chain being in the subset S and the subset S, . In sections 2 and 3 

a b 

we present a recursive formulae for evaluating these probabilities. Subse- 
quent examples show that the computation is considerably simpler than the 
classical method of solving systems of linear equations. Our method involves 
nothing but repeated substitution and is easy to perform and program even 
for a large number of states. 

The need for studying these ratios arises in problems connected 
with finite automata with binary inputs and outputs driven by a Bernoulli 
sequence. These, in turn, appear in the so-called finite memory problems 
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(References [1] through [4]), which are currently receiving considerable 
attention in literature. 

The reason for writing this report is twofold. First since the 
proofs of our formulae (Sections 4 and 5) are basically algebraic and thus 
rather long it is usually necessary to condense the proof when the formula 
is used as a lemma. Hence we wanted to have the proof documented in full 
detail for reference. Next, it is conceivable that Markov chains of the 
type studied here may be encountered in various stochastic models. Hence, 
the second purpose of this report is to provide an access to our results 
to other workers in the general area of stochastic modelling. 

To this we would like to add that the two formulae can probably be 
generalized in several directions. For instance, inspection of the proofs 
indicate that the same method could still be used to establish similar 
formulae for a larger class of chains, namely without the property (3) 
above . 

The part on ergodic chains (Sections 2 and 4) and the part on 
absorbing chains (Sections 3 and 5) can be read independently. 
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2 . Ergodic Chains . 

Let r = {r (2) ,r (3),...} and r, = {r (2) ,r, (3) , . . . } be two 
— a a a — b b b 

sequences of positive integers such that 1 £ r (i) < i, 1 ^ r, (i) < i, 

a b 

i = 2,3,... With each such pair (r , r,) we associate a class 

—a — b 



E(r ,r, ) = {M 
—a — b n ,m 



n=l , 2 , . . , ; m=l , 2 , . . . } 



of finite ergodic Markov chains. The chain M has n + m states 

n,m 

which are divided into two subsets S and S, with n and m states 

a b 

respectively . 

We label the states in S by (i,a) , i = l,...,n, and the 

a 

states in S, by (i,b) , i = l,...,m. The transition probabilities are 
b 

as follows: 



P((i,a) -> (i+1 ,a) ) = p , i = l,...,n-l, 



P ( (n , a) -* (m ,b) ) = p , 



P((i,b) + (i+l,b)) » P b , i = l,...,m-l, 



P((m,b) (n ,a) ) = p fe , 



P((i,a) (r (i) ,a) ) = q , i = 2,...,n, 

3 , 3 



P((l,a) (1 ,a) ) = q > 



P((i,b) ( r b (i) >b) ) = q fe , i = 2 , . . . ,m, 

P((l,b) -»■ (l,b)) = q b . 

Here 0<p <1, 0<p, <1, q = l- p, q =l-p. All other 

*a r b a a b b 

transition probabilities are zero. The transition diagram is depicted in 



Figure 1. 




Figure I 
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Proposition 1: Let M € E( r ,r, ) , let y(s) , s £ S US, be its 

n ,m —a — b a b 

stationary distribution, let 



u(s ) = £ y(s) and y(S ) = £ y(s) 

s€S a s6S b 



be the stationary probabilities of the chain being in S and S, 

a b 

respectively. Then 



y (s a ) 



n 



( 2 . 1 ) 



where and are polynomials in p^ and p^ respectively satisfying 

the recurrence relations 



Vi-'i 4 v L„vr‘- a i- 1 - "- 1 - 2 <2 - 2) 

£=r (n+1) 
a 

m 

Vl - »b + % , l in ¥'' m- 1,2 (2.3) 

£=r, (m+1) 
b 

Hence, both A and B have integral coefficients and are of degree 
n m 

less than n and m, respectively. 

(For the proof see Section 4.) 



Example 1: Let (r ,r 1 ) be given by the following table 

1 — n —13 



1 r (i) r (i) 

a b 

2 1 1 

3 1 1 

4 2 3 

5 • 3 

6 



6 




Figure 2 
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let n = 4, m = 5. The transition diagram of this 9-state chain is 
in Figure 3. 

First evaluate and B^. From (2.2) we have 

A 4 ’ p a + q a (A 2 p r Z+A 3 p r 3) ’ 



A 3 ' p a + q a A l p a~‘ + q a A 2 P a~ 2 • 



A 2 ' p a + ’aVa*' 1 



Aj - 1 , 



and substituting from the bottom to the top gives 



A 1 = 1 , 



A 2 = P a + q a ’ 1 > 



A 3 = P a + q a p a + q a ’ 



A. = p 3 + q p + q p 2 + q 2 p + q 2 , 
4 a a a aa a r a a 



or by substituting q = 1 - p 



A = p 3 
4 a 



p 2 + 1. 

a 



B 5 = p b + q b< B 3 p t 3+ Vr> ' 



B 4 * p b + q b fi 3 P b” 3 



b 3 - »b + q b <B i p r i+B 2 p r 2) ■ 



Similarly from (2.3) 



and again substituting 



b i = 1 , 



= 



P b + 



= 



P b + 



B , = 






B _ = 






+ q b p 



or substituting for q, = 1 - p, 

b b 



B 5 = 



Hence, from (2.1) 
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3 . Absorbing Chains . 

Let r = {r (1) ,r (2),...} and r, = {r (1) ,r (2) , . . . } be two 

3. SL 3 D D D 

sequences of nonnegative integers such that 

0 £ r (i) < i, 0 £ r (i) < i, i = 1,2,... 

3 D 

With esch such 3 pair (r ) we associ3te a class 

— 3 — b 

A(r ,r,) = {M : n=l,2,... ; m=l,2,...} 

—3 — b n ,m 

of finite absorbing Markov chains. The chain M has n + m + 1 states, 

n ,m 

two of them absorbing and the rest transient. One state is always 
designated as an initial state while the remaining n + m states are 
divided into two subsets and with n and m states respectively, 

each containing one of the two absorbing states. 



We label 


the states in 


S by 

a 


(i,a) i = 1 , . . . ,n 


with 


(n,a) 


absorbing, and 


the states in 


S b by 


B 

i — i 

II 

♦H 

rO 


with 


(m,b) 


absorbing. The 


initial state 


is labeled (0,a) or (0,b) 


or just 0 



as needed . 

The transition probabilities are as follows: 

P((i,a) -* (i+1 ,a) ) = p , i = l,...,n-l, 

a 

P((n,a) -* (n,a) ) = 1, 

P((i,b) -* ( i+1 , b ) ) = P b » i = l,...,m-l, 

P ( (m,b) - (m,b) ) = 1, 

P ( (i,a) -> (r (i) ,a) ) = q , i = l,...,n-l, 

a a 



P((i,b) -> (r b (i),b)) = q b . 



X 1 , . . . , 111*^1 , 
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p (o -*■ (1 ,a) ) 



P a + 






p (°- (i.w) ■ F ^— . 

a b 

Here 0<p <1, 0 < p, < 1, q = 1 - p , q. = 1 - p. . All other 

a b a a b b 

transition probabilities are zero. The transition diagram is depicted 
in Figure 2. 



Proposition: Let M 6 A(r ,r, ) , let Tr(a) and tt (b) be the probabilities 

n ,m —a — b 

of absorption in the state (n,a) and (m,b) respectively, if the initial 
state is the state 0. Then 



( , P n B 

tt (a) _ __a m 

tt (b) m A ’ 

p b n 



(3.1) 



where A^ and are polynomials in p^ and p^ respectively satisfying 

the recurrence relations 



n k-r (k) 

A ., = 1 - l A r ( k )P a E , A Q = 0, n = 1,2,..., (3.2) 

k 2 a 



n+1 



m k-r (k) 

Vl = 1 ' q b J, B r. (k) P b ’ ®0 = °» m = l,2,... . (3.3) 

k=2 b 



Hence both A and B have integral coefficients and are of degree 
n m 

less than n and m respectively. 

(For the proof see Section 5.) 
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Example 2 : 



Let (r , r, ) be given by the following table 
~ a — b 









1 

2 

3 

4 

5 



0 

1 

1 

2 



0 

1 

1 

3 

3 



let n = 4, m = 5. The transition diagram of this 10-state chain is 
in Figure 4. 

First evaluate A. and B c . From (3.2) we have 
4 5 



A 4 = 1 " q a (A l P a' 1+A l P a _1) ’ 



A 1 = 1 - q A nP 



2-0 



a O^a 



A 0 = 0 • 



and substituting from the bottom to the top 



A o = 0 ’ 



A = 1 , 



A. = 1 - q p - q p 2 , 
4 n a a a a 



or substituting for q = 1 - p„ 



A. = p 3 
4 a 



p a + 1 ' 



= 



= 



1 - q b (B lP 2-l+B lP 3-> + B 3 pJ-l) , 
1 - VlPb' 1 ' 



B, = 1, 



Similarly from (3.3) 
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Figure 4 








14 



and again substituting 



°i - 1 , 



B 3 ‘ 1 - p b q b • 



B 5 - i - p b q b - P 2q b - p b q b + p 2q2 , 



or 



= 2 Pu - Pu - Pu + 1- 



Hence, from (3.1) 



•n(a) 

ir(b) 



P a 2p b ~ P b ~ P b + 1 
^ p a _ P a + 1 
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4 . Proof of Proposition 1 . 

Let P be the transition probability matrix for the chain M , 

n ,m 

where the first n rows and columns correspond to states (1 ,a) , . . . , (n,a) 
and the following m rows and columns to states (m,b) , . . . , (1 ,b) . 

Let p_ = (p (1 ,a) , . . . ,y (n,a) (m,b) , . • . (1 ,b) ) be the stationary 

distribution, so that 



H(I-P) = 0 , 



( 4 . 1 ) 



where I is the identity matrix. Now partition the matrix P into four 
submatrices 



P = 



p 


V 


a 


a 


V, 


P, 


b 


b 



where P is an n x n matrix 
a 



Row: 



P = 
a 




r a (i) 



r (i.) 
a 



Column : 



1 



i 



n 
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and P, is an m x m matrix 
b 

Row: 
n 

1 

r a (1 > 

2 
1 

Column: n i r,(i) 1 

b 

Notice that each row of these matrixes contains exactly one entry q, 

th 

namely the (i,r(i)) — one, and that the labelling of rows and columns 

of P begins at the lower right corner while the labelling of P is 
b a 

the usual one (beginning at the upper left corner) . 

The off-diagonal matrices V and V, consist of all zeros except 

a b 

for the lower-left corner entry of V , which is p , and the upper- 

a a 

right corner entry of V, , which is p . 

b b 

With this partitioning the equation (4.1) decomposes into two 
equations 



^ (I - P a> ' 


(0, . . . ,0,y (m,b)p b ) , 


(4.2) 




(p(n,a)p ,0, ... ,0) , 

a 


(4.3) 




where 
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p = (p(l,a) , . . . ,p(n,a)) , 

d 

and 



P^ = (p(m,b) ,. . . ,p(l,b)) 



Consider the matrix equation (4.2) first. Solving for p gives 

a 

p^ = (0, . . . > 0,p(m,b)p b ) (I-P a ) _1 , 

or denoting a.. the (i,j)-^- entry of the inverse (1-P ) ^ 

a 



P(i»a) = p(m,b)p a , j = l,...,n. 

D n , 1 



(4.4) 



By the well-known formula for matrix inversion 



I - P 



a . . = 
iJ 



(j ,i) 



I - P 



where | I - P | is the determinant of I - P and | I - P | ( . 

a a a^kjio) 

th 

is the (k,£) — cofactor of i I - P I • Hence 

' a 1 



n y(m,b)p n 

(S a ) = l P(i.a) - | ! . p | l 
i=l 1 a 1 i=l 



I - P 



a (i >n) * 



(4.5) 



Next let A be the determinant of the n x n matrix obtained from 
n 

I - P by replacing the rr^- column by a column of l's, 
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Expanding 



along this last column we obtain 



A 

n 



n 

l 

i=l 



l I 



I - P 



(i,n) 



th 

since the (i,n) — cofactors of A and I I - P I are identical. 

n 1 a ' 

Thus, (4.5) can be written as 



y(s ) 

a 



V (m,b)p b 

“T - p 



A . 
n 



' a ' 

Next since the only flow of probability between sets 
through states (n,a) and (m,b) we must have 



S and S 
a b 



is 



p(m,b)p = y(n,a)p 
b a 



(4.7) 



in the stationary regime. Using (4.4) we obtain 

I I - P 



y(n,a) = p(m,b)p 



a (n,n) 



I - P 
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and substituting from (4.7) 



But 



I - P = p I - P L N 

a 1 *a' a 1 (n,n) 



I - P 



a 1 (n,n) 




which is same as the determinant of the (n-1) x (n-1) matrix I - P 

a 

obtained for the chain m € E (r^ ,r^) . Employing temporarily the super- 

script (n) for the number of states in S we have a recurrence relation 

cl 

i i <n) - p<"> i . p a i i'-- 1 ' - i , 



and since 



- P^ I = p we obtain 
a 1 a 



Thus 



i <n) - p (n > i . P n . 

a 1 r a 



p(S a ) = y(m,b)p b . 
P a 



Now going back to (4.3) and repeating all the steps above we obtain a 



similar expression 



20 



y(s ) = -r y(n,a)p , 
b m a 



where B is the determinant of order m 
m 



m 




- p b p b 



Hence, using again (4.7) we have 



/ p \ m 

y(S ) p A 
a _ _b n 

y(S,) n B ’ 

b p m 
a 

and it remains to prove that A and B satisfy the recurrence relations 

n m J 

(2.2), (2.3). 

We begin with the determinant A^. To evaluate this determinant 



let 



I, = (i = 2 n : r (i) = 1} . 

1 a 



Notice that is the set of exactly those row indices i for which the 

th 

(i,l) — entries in (4.6) are -q . Now multiply the first row in (4.6) 

a 

by q /p and add it to all rows such that i € I-, • The determinant 
a a x 



becomes 
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A 

n 



P -P 

0 p -p 




• -q 



0 



1 -p 




where we temporarily dropped the subscript a to ease the notation, 
entries in the last column are given by 



t 



(n) 

2i 



1 + q/p if i € I v 
1 if i £ I r 



Expanding this determinant 



along the first column we have 



D 



(n) = 
2 




where 




D 



(n) 

2 



is the original determinant (4.6) and 




The 
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is of order n - 1. Notice that the entries in the first column in 
are -q only for row indices i = 3,...,n such that either r(i) = 1 
or r(i) = 2. Hence, calling I 2 = {i = 3,...,n : r(i) < 3}, multiplying 
the first row in D^ 11 ^ by p/q and adding to rows with i £ ^ this 
determinant becomes 




(n) 

32 



(n) 

2n 



The entries in the last column are 



t 



(n) 

3i 



t 



t 



(n) 

2i 


+ ^ t (n) 
q 22 


if 


i 6 I 


(n) 

2i 




if 


i £ I 



2 * 
2 * 



Expanding again along the first column we have 



D 



(n) = 
2 




> 



where 
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Proceeding in this fashion we obtain a sequence of determinants 

h( n ) 

1 ’2 n ' 

where is of order n - k + 1. The entries t,^ in the last 

k ki 

column of satisfy the recurrence relation 

.(n) _ 



( 4 . 8 ) 



t^ n) + £ if i € I, , 

ki q kk k 



"k+l,i 



_(n) 

"ki 



if i i i k , 



( 4 . 9 ) 



where 



Further 



so that 



k = i = tj n ^ = 1, 



= {i = k+l,...,n : r(i) £ k} 



n (n) _ n (n) 
D k+1 " pD k 



(n) _ n-1 (n) _ n-1 (n) 

D - p D = p t 
1 n r nn 



( 4 . 10 ) 



Consider now the determinant of order n + 1 obtained 



by using the same sequence _r, 



A (n+1) 




-q i 
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Applying the above procedure to D^ n+ '*"* = A^ n+ ^ we obtain a sequence 



(n+l) 
J 1 

.(n+l) 



,D 



(n+l) 
n+l ’ 



(4.11) 



where the determinants again satisfy (4.9) with n replaced by 

n+l. Arrange now the last columns of the sequences (4.8) and (4.11) 
into triangular arrays as follows: 



. ( n ) 
'11 
.(n) 
"12 



.(n) 

'22 



,(n) 



. (n) (n) 






'In 



'2n 



T (n+l) 



. (n+l) 

11 

(n+l) (n+l) 
12 ’22 



(n+l) (n+l) . 

In » 2n 

(n+l) (n+l) 

l,n+l’ 2, n+l 



t (n) 

nn 



t (n+l) 
nn 

(n+l) (rri-1) 
n,n+l’ n+l, n+l 
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Since for i £ n by the definition of sets I 



(n) 



i <E if and only if i 6 



the first n rows of and are identical, i.e. 



(n) (n+1) 

ki ki 



i=k,...,n; k = 1 , . . . ,n. 



(4.12) 



Next by (4.9) 



(n+1) = 
k,n+l 



if k = 1 , . . . ,r(n+l) , 



k-1 



1 + -^ T t„„ ifk = r (n+l)+l , . . . ,n+l. 
v £=r (n+1) ’ 



In particular for k = n + 1 since r(n+l) < n + 1 



n+1, n+1 P t . r( £ +1) 



(n+1) 

n 



(4.13) 



But by (4.12) for SL < n + 1 



(n+1) _ t (n) = 



'll 



n 



. = t 



00 

ll ’ 



so that (4.13) becomes 



(n+1) m 1+ i y t (£) 
n+l>n+l P £=p( i +1) « 



D 



(n+1) 



n D 



00 



n 



= 1 + 



P 0 = 



I 



£=p (n+1) p 



S,-l 



Hence, by (4.10) 
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or calling again = A we have 

1 n 



A 11 _L 

A . i = p + q , 

n+1 a a >, n l 

l=r (n+1) 



V a n- ^ 

L A oP 



n 1 ^ 2 } • • • ) 



where clearly = 1 . 

The recurrence relation for B is established in exactly the 

m 

same fashion. 



Noticing the obvious fact that the polynomials and must 

have integral coefficients completes the proof of Proposition 1. 
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5 . Proof of Proposition 2 . 

Notice first that with 0 being the initial state any subsequent 

visit to this state is a recurrent event. Call this event E^ . Next call 

E the event which occurs if the chain after leaving the state 0 reaches 
a 

the absorbing state (n,a) without any further visit to state 0. Similarly, 

define E, for the absorbing state (m,b) . Now clearly 
b 

P(E q ) > 0, 

and since the absorbtion in (n,a) occurs if and only if we have either 



and similarly 



so that, 



Next 



E or R_E or R^R^E etc. 
a 0 a 0 0a 



it (a) 



P (E a ) 

1 - P(E Q ) » 



ir(b) 



p < E b> 

1 - P(E 0 ) 



, . P (E ) 
tt ( a) _ a 

n(b) P(E b ) 



P(E a ) 




a 

+ P 



b 



P(E>), 



(5.1) 



(5.2) 



where E 1 is the event which occurs if and only if the chain after leaving 
a 

the state (l,a) reaches the absorbing state (n,a) without ever visiting 
the state 0. 

Consider now a subchain M* obtained from the chain M^ m by 
making the state 0 an absorbing state and deleting states (l,b) though 
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(m,b) . The transition probability matrix for this subchain is the 
(n+1) x (n+1) matrix 




If this subchain is started at the state (l,a) then PCE 1 ) is 

a 

equal to the probability of absorption in (n,a) for this subchain. Using 
the well-known result from the algebraic theory of Markov chain (cf. [5], 
Theorem 3.3.7) we have 



-1 



P(E i ) ' Pa“l,n-X' (5 ' 3) 

and Q is the (n-1) * (n-1) matrix of transi- 



where [a..] = ( I — Q ) 
xj a 

tion probabilities between transient states of i.e. 
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By the formula for matrix inversion 

I i - Q 



l.n-l | I - Q, 



a 1 (n-1,1) 



(5.4) 



where | I - Q | is the determinant of I - Q and 

cL cl 

, St 



is its (n-1,1) — cofactor. Now 



I - Q 



a 1 (n-1,1) 



I (n-1,1) ' (_1> 




n-2 

= P a » 



and calling = 



I - Q I we obtain from (5.2), (5.3) and (5.4) 
a 1 



P( V ■ p + p, A 

r a b n 



(5.5) 
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Following the same procedure for the event E we obtain 



m 



P( V 

a b m 



(5.6) 



where | I | and is the (m-1) x (m-1) matrix of transi- 

tion probabilities between transient states (l,b) , . . . , (m-l,b) . Hence, 
(5.1), (5.5) and (5.6) yields (3.1) and it remains to establish the recur- 



rence relations for A and B . 

n m 



To do this we evaluate the determinant 



aii n r _ 




where we dropped the subscript a to ease the notation. Notice that 
is of order n - 1 and that each row but the first has either exactly one 
subdiagonal entry equal to -q, namely the (i,r(i))— , or all subdia- 
gonal entries are zero. The former case occurs if r(i) > 0 while the 
latter if r(i) = 0. 

Next consider the determinant A , of order n obtain for the 

n+1 

chain M with the same (r ,r, ). Then 

n+1 ,m — a — b 
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n+1 





0 


A 




n 


l 




1 0 
1 




1 

,- p 


-q 


1 1 



Now if r (n) = 0 then there is 

3 . 



no -q in the last row and hence 



A = A . 
n+1 n 



(5.7) 



> ^ then expanding A^ a l° n g the last column gives 



A - = A + pD , 
n+1 n 



(5.8) 



where 



n-1 



-q 



is of order n-1. Notice that the entry -q in the last row moved one 
step to the right. Expanding again along the last column gives 



D n = pD n l 5 

n n-1 



where D is of order n - 2 i 
n-1 
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1 0 
1 1 
1 


D , 


1 

1 

A . * 


n-1 


n 2 o 




1 

~P 




-q 10 



Now repeating this the entry -q eventually (after n - r (n) 

cL 

reaches the diagonal and we have 



D r (n)+l 
a 



0 



A r (n) 
a 



0 . . . 



I 

I 




0 '-q 



> 



and expanding this determinant along the last row yields 



D r (n)+l qA r (n) * 



Substituting back into (5.8) we obtain 



, . n-r(n) , 

A ,. = A - qp A , . 

n+1 n r (n) , 

a 



which holds for any n = 1,2,... such that 



r (n) > 0. 
a 



steps) 



(5.9) 
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To include the case r (n) = 0 define A n = 0. Then (5.9) reduces to 

a u 

(5.7). Finally, repeatedly substituting for in (5.9) and using the 

obvious fact = 1 we obtain the recurrence relation (3.2). Notice 

that (3.2) holds also for n = 1 since r^(l) = 0 always. 

The relation (3.3) for B is established in exactly the same 

m 

fashion from | I - | . 

Noticing the obvious fact that the polynomials A^ and must 

have integral coefficients completes the proof of Proposition 2. 
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